test: verify real Whisper inference on NVIDIA T4 - #9

Verified on an NVIDIA T4. All 8 real-model tests passed with faster-whisper base.en on CUDA FP16 (Colab Tesla T4, commit 096a034). The recorder reported complete: true with no problems. The evidence is committed unmodified in docs/benchmarks/whisper-t4/. This is one verified configuration; see the limitations below.

Problem

The offline suite (fake and gated backends, run in CI) is extensive, but the only real-model test, tests/test_whisper.py, loaded base.en on CPU with int8, decoded silence, and checked that the result was a string. It did not check CUDA, FP16, transcript correctness, error handling, the server, or the WebSocket path, and it had not been confirmed as executed in the current verification cycle.

Implementation

tests/test_whisper.py: 8 tests, marker whisper, still deselected by default. Each level claims only what it exercises.

Fixtures and thresholds:

• Audio. The speech tests need python -m benchmarks.prepare_audio --num 5 and check each file's PCM SHA-256 against the recorded Phase 1 precision run, so WER is computed on the audio whose FP16 WER was recorded.
• Thresholds are regression guards, not measurements. Recorded T4 FP16 values on that audio are 0.0708 corpus WER (8/113, precision_t4.json) and 0.1154 for the streamed utterance (3/26, gpu_smoke.json).
• Device selection. ECHORELAY_TEST_DEVICE=cuda selects the GPU. A CUDA request with no visible device fails; it does not skip.

benchmarks/whisper_verification.py (new, standard library + existing helpers): runs python -m pytest -m whisper -rA -s tests/test_whisper.py and writes record.json, pytest.log and junit.xml. The record holds:

• UTC start and finish times, and the git commit with its dirty flag
• full nvidia-smi output, GPU and driver
• Python and package versions
• model, device and compute type
• the exact command
• any PYTEST_ADDOPTS / PYTHONPATH / ECHORELAY_MODEL_DIR overrides, so a stubbed run cannot pass for a real one
• per-test outcomes, with recorded WER as JUnit properties

complete requires, and problems lists every violation of:

• exactly the 8 expected tests (by name) passed, with 0 failures, errors and skips, and pytest exited 0
• the identity the loaded runtime reported during the run matches the request, rather than trusting the recorded settings:
  • faster_whisper.transcribe.WhisperModel over ctranslate2._ext.Whisper
  • the CTranslate2 device and compute type (exactly float16 on CUDA)
  • at least one CUDA device
  • the models--Systran--faster-whisper-base.en snapshot faster-whisper resolved, with the model.bin SHA-256
  • an English-only model
• nvidia-smi lists a T4 for CUDA (--expect-gpu)
• no PYTEST_ADDOPTS / PYTHONPATH override, and a clean checkout

tests/test_whisper_verification.py (32 offline tests, run in CI) covers each rule.

tests/test_observability.py: macOS flake fix.

• What failed: test_runner_counts_timeouts_while_queued (CI run 40155745468, attempt 1: in_flight was 1 after await asyncio.sleep(0.4)).
• Cause: a timing assumption, not a cleanup bug. The timed-out decode keeps its slot until its 0.5 s worker thread returns (documented runner behaviour), and the slot is then released on the loop. So the fixed sleep left ~100 ms of slack. With a simulated extra wake-up delay, the old assertion failed at 110 and 150 ms, while the slot was still released 10–53 ms later with free == 1. The scheduler correctly skips the timed-out waiter.
• The fix:
  • The test uses GatedTranscriber, so the test decides when the decode finishes.
  • It asserts the slot is held while the decode runs, waits for the release with a bounded until(), and then asserts in_flight == 0, waiting == 0, free == 1, and that the queued request never ran.
  • runner.shutdown() is in a finally.
• Kept and verified: the original assertions are kept. The test passes with a 300 ms simulated slow wake-up, and it still fails if the slot is never released (bounded wait fails after 5 s) or is handed to the timed-out waiter.

Evidence and docs (ee4b7f3).

• docs/benchmarks/whisper-t4/raw/: record.json, pytest.log and junit.xml, committed byte-identical to the returned ZIP, with SHA256SUMS.
  • verify_evidence now checks 4 manifests and 173 files.
  • The files total about 11 KB; no weights, audio or credentials are committed.
  • The only host-specific values are Colab paths and an ephemeral container hostname in junit.xml.
• docs/benchmarks/whisper-t4/README.md: the result, provenance, the independent checks below, and comparisons with earlier T4 evidence.
• docs/whisper-verification.md: the claims of each test, verified status, the filled verification log, limitations, and the reproducible two-cell Colab procedure.
  • It also records the troubleshooting incident, taken from the operator's account of the session. The failed attempts' output is not committed.
• README.md:
  • The GPU-pending statement is replaced with the verified result.
  • Offline test counts are now 269 per configuration with 8 real-model tests deselected (previously 209 with 1 deselected).
  • The evidence inventory now lists 4 manifests and 173 files, and docs links were added.
• benchmarks/COLAB.md:
  • The stale "nothing has been run on a GPU" statement is gone.
  • Clones now use main instead of the old feat/streaming-mvp branch.
  • The observed cuBLAS 12 issue is documented, and stage 2b runs the recorder without Colab's PYTHONPATH.
• .gitignore: a narrow exception so the evidence pytest.log can be committed, like the existing one for server.log.

Unchanged:

• inference, scheduling and the WebSocket protocol
• existing benchmark data, results and manifests
• WER thresholds and verification criteria
• dependencies and the CI workflow
• no LICENSE

tests/test_whisper.py, benchmarks/whisper_verification.py and src/ are byte-identical to 096a034, the verified commit.

tests/test_units.py: second flake fix (0858899).

• What happened: on ee4b7f3, a docs-only commit, the macOS job failed once more and passed on rerun.
• The suspect: test_runner_releases_slot_only_when_thread_finishes has the same defect class as the flake fixed in 096a034. It sleeps a fixed 0.4 s after a timed-out 0.3 s decode, then asserts the slot was released, leaving about 200 ms of slack.
• Reproduced: with the worker thread returning 0.25 s late, it fails with assert 1 == 0 on in_flight.
• The fix:
  • It now waits on state with a bounded until(), keeps every assertion, and always shuts the runner down.
  • It passes the slow-wake simulation and 30/30 repeats, and still fails if the slot is never released.
• Caveat: that run's log isn't readable from this environment, so this is the likeliest match for the failure, not a confirmed one.

Real-model verification (T4)

Independent checks on the evidence. These were made on the files themselves, without trusting the recorder's verdict:

• junit.xml re-parsed reproduces record.json's results exactly.
• The validator, re-run on the raw inputs, reports no problems.
• All 8 JUnit test line numbers match the test definitions in tests/test_whisper.py at 096a034, so the suite that ran is the committed one.
• pytest.log agrees, and its timestamps fall inside the run window.
• All five transcripts are word for word identical to the Phase 1 FP16 hypotheses (precision_t4.json). That run used the same GPU, driver and library versions.
• 0.0531 equals the v2 WER committed in Phase 3 for the same five files.

Correction. The suite cited Phase 1's 0.0708 as the recorded baseline, but that was scored with the v1 normalizer. The comparable v2 value is 0.0531, and the docs now say so. The comment in tests/test_whisper.py still says 0.0708, left as is so the verified file stays byte-identical. The bounds themselves are unchanged.

Troubleshooting incident (operator's account; output not committed).

1. First attempt failed: Library libcublas.so.12 is not found or cannot be loaded.
   • Colab exposed CUDA 13's libcublas.so.13, with LD_LIBRARY_PATH=/usr/lib64-nvidia.
   • The pip-installed libcublas.so.12 lived under dist-packages/nvidia/cublas/lib, outside that path.
2. Fix: that directory was prepended to LD_LIBRARY_PATH, plus cuDNN 9's directory if present.
3. Second attempt: all 8 tests passed, but the recorder correctly reported NOT COMPLETE because of Colab's default PYTHONPATH=/env/python. The check was not relaxed.
4. Final run: with PYTHONPATH removed from the recorder's environment, it exited 0 and reported complete: true.

LD_LIBRARY_PATH is not captured by the record, so the fix appears only in this account.

Offline results (exact)

Local, Linux, no GPU, at 0858899. Apart from tests/test_units.py, the code is identical to 096a034; the rest of the change is docs and evidence.

Earlier, at 096a034:

• test_observability.py and test_whisper_verification.py passed 30/30 repeats on both versions.
• ruff check --select F,E9 was clean on new and changed files.

The repo has no configured lint or type checker. 269 = 237 existing tests + 32 validator tests.

Pre-GPU harness checks (not model evidence): a stub WhisperModel exercised the harness and recorder; the results are described in docs/whisper-verification.md.

Reproducing

Use the two Colab cells in docs/whisper-verification.md. They include the library-path fix and run the recorder without PYTHONPATH.

Remaining limitations

• One verified configuration: base.en, CUDA FP16, one Colab T4, one run.
  • Not verified: CPU int8, int8_float16, other model sizes, other GPUs, other driver or library versions.
• The recorder misses some run conditions.
  • It does not record LD_LIBRARY_PATH, which was required on Colab.
  • pytest auto-loaded the image's installed plugins (typeguard, anyio, langsmith, platformdirs). They appear only in pytest.log. The integrity check covers overrides, not installed entry-point plugins.
  • These are reported, not changed: changing the recorder would make the branch differ from the verified code.
• Not covered by the suite:
  • GPU memory release (only worker threads and the port are checked)
  • concurrent load (the benchmarks cover it)
• CI does not run the real-model tests. Re-verification is manual, on Colab. CI does test the recorder's validation rules.
• The model.bin hash has nothing to compare against. It is recorded for traceability; no independent reference hash exists in the repo.
• macOS CI flakes:
  • Fixed: test_runner_counts_timeouts_while_queued (096a034, confirmed from the log).
  • Fixed: test_runner_releases_slot_only_when_thread_finishes (0858899, the likeliest match, not confirmed).
  • CI on 0858899 passed 10/10 on the first attempt (run 40163113395).
  • Other tests that pair a fixed sleep or a wall-clock bound with an assertion haven't been fully audited, for example in test_lifecycle.py, test_reliability.py and test_live_server.py.

https://claude.ai/code/session_017DfLG9q5h7eiawGNwaBswq

Generated by Claude Code